Hooking applications are becoming increasingly popular, especially in the security and network management arts. Such hooking applications are adapted for hooking various aspects of an interface. By way of example, some of such applications are capable of hooking application program interface (API) calls.
Such API hooking is a technique where an API is modified so that subsequent invocations of a particular function transfers control to a handler. This handler may then, in turn, analyze the original API invocation, report usage of the API, modify relevant parameters, etc. Further, in the case of security applications, API hooking may serve to enable a decision as to allowing or disallowing the original API to be invoked.
As the foregoing hooking applications become more and more prevalent, there is an increasing likelihood that more than one hooking application may reside on a single system. Unfortunately, hooking applications have historically been unable to coexist.
This scenario of multiple co-existing hooking applications (e.g. a first hooking application and a second hooking application, etc.) presents various problems. For example, the second hooking application, as part of its handling of API interceptions, typically invokes what it perceives as the original API upon completion of handling. If, however, the first hooking application was already hooking the API when the second hooking application started, the second hooking application will invoke the first hooking application.
This presents a particular problem if the first hooking application relies on a return address to transfer control to a handler of the first hooking application. Normally, this return address points back to the original API and results from the transfer of control from the original API to the first hooking application handler. However, if the second hooking application invokes the first hooking application, the return address points, instead, into the first hooking application. One common reason to use this return address is to identify which API was originally invoked to allow reuse of a common interception handler for all hooks. Unfortunately, if such return address points back to the first hooking application, the use of the return address in the abovementioned scenario results in failure.
Another problem concerns the situation where the first hooking application unloads. In this case, the second hooking application, while attempting to invoke what it perceives as the original API, transfers control to an address that is not valid (since the first hooking application is unloaded). To this end, the instant process, or even the entire system, is subject to failure.
There is thus a need for overcoming these and/or other problems associated with the prior art.